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Abstract 

Mission Operations and Command Assurance 
(MO&CA) is a Total Quality Management (TQM) 
task on JPL projects to instill quality in flight mission 
operations. From a system engineering view, 
MO&CA facilitates communication and problem- 
solving among flight teams and provides continuous 
process improvement to reduce the probability of 
radiating incorrect commands to a spacecraft. The 
MO&CA task has evolved from participating as a 
member of the spacecraft team, to an independent 
team reporting directly to flight project management 
and providing system level assurance. JPL flight 
projects have benefited significantly from MO&CA's 
effort to contain risk and prevent rather than rework 
errors. MO&CA's ability to provide direct transfer of 
knowledge allows new projects to benefit from 
previous and ongoing flight experience. 
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1. Introduction 

A long-term program is in progress at JPL to reduce 
cost and risk of flight mission operations through 
defect prevention and error management. Flight 
mission operations require systems that place human 
operators in a demanding, high risk environment. 
This applies not only to mission controllers working 
in the "dark room" and Deep Space Network (DSN) 
operators configuring and monitoring DSN 
operations, but also to flight teams that plan the 
mission and develop the command sequences and to 
engineering teams responsible for analyzing 
spacecraft performance. The flight operations 
environment generally requires operators to make 
rapid, critical decisions and solve problems based on 
limited information, while closely following standard 
procedures (Refs. 1-3). The mission operations 
environment is, therefore, inherently risky because 
each decision made is potentially mission critical. 


To contain this risk at JPL, flight mission operations 
procedures (as described in Refs. 4-5) currently 
require intensive human reviews. In addition, when 
an error does occur, rapid rework is required to 
ensure mission success. This strategy has worked 
well to reduce risk and has ensured the success of 
JPL missions. However, the large human labor 
investment required for review and rework has 
substantially contributed to the overall cost of flight 
mission operations. Prevention of errors would 
greatly reduce both cost and risk of flight projects. 
The motivation of the long-term defect 
prevention/error management program is to contain 
risk in a more cost effective manner by preventing 
errors rather than reworking them. The goal of this 
program is the management, reduction and prevention 
of errors. 

A major element of this program is the Mission 
Operations and Command Assurance (MO&CA) 
function. MO&CA provides a system level function 
on flight projects to instill quality in flight mission 
operations. MO&CA's primary goal is to help 
improve the operational reliability of projects during 
flight. MO&CA’s effort focuses directly on 
continuous process improvement to reduce the 
probability of radiating incorrect commands to a 
spacecraft. MO&CA occupies a unique position in 
the flight project organization, reporting to both flight 
project management and the System Assurance 
Division of the JPL Office of Engineering and 
Review. As a result, MO&CA is able to cross 
operational boundaries between teams and offices on 
a flight project. This system level view enables 
MO&CA to enhance inter-team communication to 
facilitate problem solving within the project. 

This paper describes the development and evolution 
of the MO&CA function at JPL and the benefits 
provided to flight projects by MO&CA. 

2. Evolution of the MO&CA Task 

The MO&CA task began on the Voyager (VGR) 
project in 1985. In response to an increase in 
command related problems a study was conducted by 
the JPL Office of Engineering and Review to review 
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the adequacy of procedures, operations and software 
involved in real-time commanding with the goal of 
reducing errors. Incident Surprise Anomaly (ISA) 
reports, problem reports written by flight team 
members when an anomaly occurs in flight 
operations, were analyzed from a seven year period 
(1978 - 1985) to determine the causes of command 
errors. Based on this analysis, recommendations 
were made to the project for improvements in flight 
procedures including: upgrading the command 
development software to improve readability of 
command printout thus facilitating command reviews 
and approvals; providing traceability between 
command forms and ISAs to facilitate analysis and 
correction of command incidents; reducing real-time 
commanding by improving the coordination of real- 
time and sequence commanding and including as 
many commands as possible in command sequences; 
and updating flight team training to include command 
awareness issues making flight team members aware 
of potential command problems and how to avoid 
them. Command development procedures were 
updated to incorporate these recommendations. 

When an opening occurred in the spacecraft team the 
following year, the position was filled by a MO&CA 
engineer who became the Systems Lead for real-time 
commanding for the VGR Project. MO&CA also 
continued to analyze ISA reports and make 
recommendations for continuous improvement to the 
commanding process. 

MO&CA provided both a system engineering 
function for the spacecraft team and a systems 
assurance function for the VGR Project. Although 
not included in the project at launch, the MO&CA 
function was successfully integrated into the 
spacecraft team. This placement allowed MO&CA to 
not only recommend changes to command 
procedures, but also to implement these changes, with 
project management concurrence, for the spacecraft 
team. 

Following VGR, a MO&CA team was activated on 
the Magellan (MGN) Project just two months prior to 
launch, on May 4, 1989. The main MO&CA task for 
MGN was to detect, analyze and correct defects that 
existed in flight operations and procedures. Tasks to 
detect and analyze defects included: review of flight 
operations processes and documentation; 
participation in operational readiness training 
exercises; assessment of command process 
compliance with requirements and procedures; 
analysis of ISA reports; and participation in software 
testing. Activities associated with defect correction 
included: command awareness training; real-time 
support to the command development process; 
participation in the analysis and correction of 


command incidents; implementation of an Anomaly 
Reporting and Tracking System (ARTS); and support 
for investigation and development of workarounds 
for hardware failures. 

One of the major efforts of the MGN MO&CA team 
was assisting the flight project to upgrade the real- 
time command process and related operational 
procedures. The initial real-time command process in 
place at launch included only a handful of steps. 
Commands were initiated within one subsystem of 
the spacecraft team, passed to another subsystem for 
coding, and submitted to project management for 
approval to transmit. No system level effort for 
review or coordination was in place. 

In the first few months following launch the real-time 
command system was in constant use due to several 
hardware problems on the spacecraft. Extensive 
operational workarounds were required to 
compensate for these problems. Because of the 
anomalies and a lack of coordination in the real-time 
command process, command problems occurred. 
MO&CA recommended improvement to the 
command process which included: review of 
commands by all subsystems prior to development; 
system level coordination of all commanding; 
management approval prior to command 
development; traceability of commands from initial 
request to final approval for transmission; 
development of rigid test requirements for all 
commands; required representation by all flight teams 
at command review and approval meetings; 
spacecraft team support of the command coordination 
meeting; training for all flight team members with the 
newly developed command procedures. 

By December, 1989, an updated real-time command 
process was in place on MGN. This well-structured 
process enabled the flight team to function well as a 
unit and respond quickly to anomalies. Real-time 
command incidents decreased dramatically despite 
the fact that the flight team continued to face 
spacecraft anomalies. 

In contrast to the VGR project, the MGN team was 
not integrated directly into an existing team on the 
flight project, but instead formed an independent unit. 
While this enabled the MO&CA team to maintain a 
systems view of flight operations, it did not provide 
the same ability to implement changes. MO&CA 
instead provided recommended changes based on 
ISA analysis and direct participation in working, 
review and approval meetings. The flight team, 
directed by project management, implemented the 
changes to operations procedures and processes. 



Due to the success of the first MO&CA teams, the 
MO&CA function was proposed to the management 
of the TOPEX/POSEIDON Project. The concept was 
well received by the Project and in November 1991, a 
MO&CA engineer was placed on staff to the 
TOPEX/POSEIDON Flight Operations System 
Manager to facilitate development of the Real-time 
Command Process. 

A unique aspect of the TOPEX/POSEIDON Project 
is the amount of planned real-time command activity. 
Due to the nature of the mission, 
TOPEX/POSEIDON is very dependent on real-time 
command activity. For example, tape recorders are 
played back via real-time commands three times per 
day. Once in place on the project, MO&CA quickly 
assessed existing flight operations plans and noted 
that an additional process for the development and 
approval of unplanned real-time commands was 
required. 

MO&CA worked with the flight team to define 
required interfaces for the unplanned real-time 
command process and develop the necessary 
procedures and process descriptions. While the 
flight teams were preparing individual team operating 
procedures, MO&CA was able to provide a system 
level overview and develop the additional process 
and procedures that cross team and division 
boundaries. As the project planned to use the real- 
time command process extensively, MO&CA 
coordinated the development of a Real-time 
Command Library. This library consisted of all pre- 
defined real-time command files that were planned 
for repeated utilization throughout the life of the 
mission. 

In addition to the Real-time Command Process and 
Library, MO&CA also developed the Operations 
Coordination Process, Anomaly Reporting Process 
and aided in development in the Configuration 
Management Process. As a portion of the Operations 
Coordination Process, MO&CA documented the 
types and level of necessary Flight Operations 
meetings. 

By January 1992, the TOPEX/POSEIDON Real-time 
Command Process and Library were in place. The 
transition to test and training activities went 
smoothly. The flight team readily adapted to, and 
exercised the Real-time Command Process, the Real- 
time Command Library, and the Anomaly Reporting 
Process. With a functioning pre-launch Anomaly 
Reporting Process for ISA reporting, MO&CA was 
able to gather and present detailed statistics to the 
project on recurring problem areas encountered 
during test and training. 


Since launch on August 10, 1992, MO&CA's efforts 
have been focused on coordination of the real-time 
command process, coordinating approval and uplink. 
MO&CA also plays a key role in sequence review 
and validation of all planned real-time commands. 

The most beneficial portion of the Real-time 
Command Library proved to be the Contingency 
Commands. The Contingency Commands were 
invaluable when spacecraft anomalies occurred early 
in the mission, facilitating recovery operations during 
a high activity period. The value added by the 
MO&CA Real-time Command Library is also visible 
daily during mission operations. The majority of 
planned real-time commands in the 
TOPEX/POSEIDON Sequence of Events are pulled 
"off-the-shelf from MO&CA’s Real-time Command 
Library. 

The TOPEX/POSEIDON Project experienced an 
immediate benefit through the direct transfer of 
MO&CA's knowledge and experience from previous 
projects. The MO&CA task combined elements from 
the VGR and MGN MO&CA experience. Like the 
MGN MO&CA team, TOPEX/POSEIDON MO&CA 
functions as an independent unit, and, like VGR 
MO&CA, TOPEX/POSEIDON MO&CA has the 
ability to implement improvements in flight 
operations procedures. TOPEX/POSEIDON 
MO&CA was the first MO&CA team to be in place 
on the flight project an extended time prior to launch. 
The team was therefore able to implement "lessons 
learned" and process improvements early while flight 
operations procedures were being developed. This 
opportunity allowed MO&CA to instill quality into 
the flight procedures in a pro-active manner, rather 
than work reactively to update processes and 
procedures after completion of mission operations 
development. 

3. Benefits of MO&CA 

The MO&CA function has evolved from 
participating as a member of one flight team on the 
VGR project; to an independent unit providing 
assessments and recommendations based on a 
systems view to the MGN project; and finally to its 
current state as a combination of both direct flight 
team participation and system level assurance on the 
TOPEX/POSEIDON project. MO&CA's unique 
position, with both a high level systems view and the 
direct ability to develop and implement changes as 
necessary, enables MO&CA to provide flight projects 
the benefits of both the assurance and system 
engineering functions. 

Originating from the Systems Assurance Division at 
JPL, MO&CA provides a direct transfer of 


663 



knowledge between current missions in addition to 
providing valuable "lessons learned" experience to 
new flight projects. New projects are able to thus 
benefit directly from both previous and ongoing 
mission operation experience. Lessons learned can 
be incorporated early on, into project requirements, 
thereby eliminating the amount of necessary rework 
on flight operations procedures. The real-time 
command process and library on the 
TOPEX/POSEIDON project are examples of this 
direct transfer of knowledge. 

Process improvement activities require the ability to 
measure and evaluate a process. MO&CA teams 
collect and analyze error data from the ISA reports 
written by flight teams on operational problems. 
Many of MO&CA recommendations for process 
improvement are based on these reports. This error 
analysis results in improvements not only to the 
project that wrote the report, but also to other flight 
projects via transfer of knowledge. The error analysis 
information is also used for analysis in the overall 
defect prevention / error management program. 

MO&CA’s unique position as an independent unit in 
the flight project organization provides its ability to 
facilitate communication and problem solving. 
Problems that span many teams and offices within a 
flight project can be effectively addressed by 
MO&CA. Coordinating real-time command 
processes is an example of this task. Flight project 
members who are faced with problems that impact 
several teams, often bring the issue directly to the 
MO&CA engineer when they cannot be addressed 
solely by their team. MO&CA is also able to 
improve the efficiency of data reporting that crosses 
team boundaries. The newest MO&CA team on the 


Mars Observer (MO) project noted that identical data 
was being reported by several of the flight teams, and 
that the data was often erroneous if 
it originated in one team and was reported by another. 
MO MO&CA worked with the teams to eliminate 
this duplication and ensure correct data was reported. 


4. MO&CA and TQM 

The MO&CA function is one example of a Total 
Quality Management (TQM) process at JPL. 
Specifically, MO&CA embodies the TQM principle 
of continuous process improvement (CPI) in which 
processes are continuously examined and analyzed 
for opportunities for improvement. Figure 1 shows 
how MO&CA implements CPI in two ways. First, 
within ongoing projects, the flight mission operations 
environment is established and MO&CA participates 
as a team member. In the course of day-to-day 
operations, anomalies are documented as ISA reports. 
The IS As then serve as data which is analyzed by 
MO&CA engineers for process improvement 
opportunities. When these opportunities are 
identified, MO&CA provides reports and data to 
support recommendations for improvement to project 
management. Finally, based on management 
approval, MO&CA helps the project implement the 
changes back into the day-to-day mission operations 
environment. This technique was successfully 
implemented on the VGR, MGN and 
TOPEX/POSEIDON projects. 

The second way in which MO&CA implements CPI 
on JPL projects is on new projects or upgrades to 
existing projects. The recommendations that are 
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developed from the data analysis on ongoing projects 
are used as input to system requirements on 
newprojects. This allows new projects to benefit 
from improvements made on past projects as 
TOPEX/POSEIDON benefited from the experience 
gained on VGR and MGN. 

Using this technique, not only do ongoing projects 
continuously improve, but each new project starts 
with a better set of requirements and better processes 
than the last one. At JPL this continuous 
improvement feedback loop has improved flight 
mission operations processes from the Voyager 
Project, to the Magellan Project, and to the 
TOPEX/POSEIDON project. Additionally, this 
continuous process improvement reduces cost and 
risk of flight mission operations. 


5. The Future of MO&CA 

Future flight missions at JPL will have smaller 
spacecraft and flight teams (Refs. 6-7). Development 
times will be reduced and the teams that design and 
build the spacecraft will also staff the flight mission 
teams. MO&CA will need to evolve to adapt to this 
ch a ngin g flight operations environment 

MO&CA will continue to provide both system 
assurance and engineering assistance to operations. 
The MO&CA function will be implemented during 
project development so that MO&CA can assist in 
developing operational procedures and participate in 
flight team training. This will streamline procedure 
development and eliminate late changes and upgrades 
thus reducing rework and cost. 

Automation of data tracking and analysis by 
MO&CA help to make operations process monitoring 
and error analysis more efficient and timely. With 
automation, MO&CA will be able to address problem 
areas quickly. Finally, ISA data will be used in a 
parallel error analysis study. The findings of this 
study (Ref. 8) will enable prevention of errors 
through improved requirements development on new 
projects. 


6. Summary 

JPL flight projects have benefited significantly from 
MO&CA's effort to contain risk and prevent rather 
than rework errors. MO&CA’s ability to provide 
direct transfer of knowledge allows new projects to 
benefit from previous and ongoing flight experience. 
The system level view of project operations provided 
by MO&CA enhances communication to facilitate 
problem solving within a flight project. 


MO&CA will continue to evolve to meet flight 
project needs. Early involvement with developing 
projects will ensure that quality is incorporated into 
mission operations during operations development 
and training. 

The MO&CA function at JPL has built quality into 
mission operations, enabling flight teams to operate 
efficiently and effectively in a dynamic space flight 
operations environment. Since MO&CA, as a TQM 
effort, focuses on continuous improvement of 
processes and elimination of rework, MO&CA will 
continue to provide benefits to flight projects. 
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